OpenLineage : Add core lineage service scaffolding by iting0321 · Pull Request #4705 · apache/polaris

iting0321 · 2026-06-11T13:17:26Z

Please noted that this PR is first out of the three and still depends on #4826 and #4827

OpenLineage : Add core lineage service scaffolding #4705 Current one
OpenLineage: add Persistence Layer SPI #4826 (second dependent one)
OpenLineage: Add Persistence schema and JDBC models #4827: (third dependent one)

Description

This PR introduces the core query/response models, a LineageService boundary, runtime service/config wiring, and disabled-by-default feature guards. It does not add persistence, ingest parsing, REST endpoints, RBAC, or forwarding.

Changes

Add polaris-core lineage query/response models for query requests, graph responses, nodes, node data, field mappings, direction,granularity, and node type in polaris-core/.../lineage/.
Add LineageService query boundary in polaris-core/.../lineage/.
Add runtime DefaultLineageService and LineageConfiguration in runtime/service/.../lineage/.
Add ENABLE_LINEAGE feature flag, defaulting to disabled, in polaris-core/.../config/FeatureConfiguration.java.
Add generated config docs for polaris.lineage.enabled in site/content/in-dev/unreleased/configuration/config-sections/.
Add unit tests for disabled guard behavior and enabled-but-not-implemented placeholder behavior in runtime/service/.../DefaultLineageServiceTest.java.

Out of Scope

Persistence SPI, persistence models, JDBC schema, and storage implementation.
OpenLineage ingest parsing and dataset/column edge extraction.
API endpoints.
RBAC privileges and authorization filtering.
Downstream forwarding and passthrough proxy behavior.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Introduces a new lineage service boundary and supporting models/configuration, gated behind static and realm feature flags, with a placeholder persistence implementation.

Changes:

Adds core lineage API models (ingest/query requests, graph/node/edge types) and SPI (LineagePersistence, LineageService).
Implements DefaultLineageService with enablement checks (static config + realm feature + persistence flag) and adds a disabled persistence placeholder.
Adds unit tests for enablement gating and persistence delegation, and introduces a new realm feature flag ENABLE_LINEAGE.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
runtime/service/src/test/java/org/apache/polaris/service/lineage/DefaultLineageServiceTest.java	Adds unit coverage for lineage enablement checks and delegation to persistence.
runtime/service/src/main/java/org/apache/polaris/service/lineage/LineageConfiguration.java	Adds SmallRye config mapping for lineage feature and persistence toggles.
runtime/service/src/main/java/org/apache/polaris/service/lineage/DisabledLineagePersistence.java	Adds placeholder `LineagePersistence` that fails fast when invoked.
runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java	Adds request-scoped lineage service implementation and enablement gates.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageService.java	Defines a core service boundary for lineage operations.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageQueryRequest.java	Adds request model for lineage graph queries.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineagePersistence.java	Adds persistence SPI contract for lineage backends.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageNodeType.java	Adds enum for node kinds in lineage graphs.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageNode.java	Adds node model for lineage graph responses.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageIngestRequest.java	Adds normalized ingest payload model.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageGraph.java	Adds normalized lineage query response model.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageGranularity.java	Adds enum for dataset vs column granularity queries.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageFieldReference.java	Adds model for dataset field references used in column edges.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageFieldMapping.java	Adds mapping model for column-granularity responses.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageEdge.java	Adds dataset-to-dataset edge model.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageDirection.java	Adds enum for query direction.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageDataset.java	Adds dataset identity model for lineage.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageData.java	Adds response metadata wrapper for datasets.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageColumnEdge.java	Adds column-level edge model.
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java	Adds realm feature flag `ENABLE_LINEAGE`.

adnanhemani · 2026-06-12T03:34:28Z

Thanks for this @iting0321! I think we are trying to, however, do too much in this PR itself. Can we remove the Persistence-related models, as we may need a bit more time to close consensus on those bits? I understand that there will be no callers of these models as a result, but we will still need this in case of both persistence and passthrough models regardless. Given that nothing in here would be considered a "public interface" IMO, we should be ok to change it later down the line, if needed.

iting0321 · 2026-06-12T15:40:59Z

Thanks for this @iting0321! I think we are trying to, however, do too much in this PR itself. Can we remove the Persistence-related models, as we may need a bit more time to close consensus on those bits? I understand that there will be no callers of these models as a result, but we will still need this in case of both persistence and passthrough models regardless. Given that nothing in here would be considered a "public interface" IMO, we should be ok to change it later down the line, if needed.

I just removed the Persistence-related models (including ingest model).
This PR now only keeps the neutral lineage query/response models, the LineageService query boundary, runtime config/service wiring,the disabled-by-default ENABLE_LINEAGE guard, and the related tests/docs.
I also updated the PR description to explain what is out of scope. Thanks!

dimas-b · 2026-06-19T19:18:25Z

+import java.util.OptionalLong;
+
+/** Dataset metadata returned in a lineage query response. */
+public record LineageData(


These new classes do not appear to be required / used by the old polaris-core code.

In the spirit of modularization, I'd like to propose defining them in a separate gradle module / jar.

Are there any plans for using these new classes in old polaris-core code?

Thanks for your feedback! I don’t currently see these classes being consumed by existing polaris-core code paths, they appear to be used only by the new lineage service boundary. So I agree they can be moved into a separate lineage module unless we expect core components to depend on them soon.

snazy · 2026-06-22T11:33:02Z

I am not convinced this should merge before we know which implementation contract it is serving.

A core lineage query/service model without a working implementation risks freezing API concepts before we have validated the actual persistence/query semantics. In particular, the local-store discussion is still open around reduced graph semantics, “latest” edge replacement, auth-filtered reads, and backend capability boundaries.

If this PR is intended only as neutral scaffolding, I think it should be very explicit about what is not being committed here, and avoid introducing model/API shapes that later persistence PRs depend on as settled. Otherwise we should probably review this together with the first implementation that proves the model is sufficient.

iting0321 · 2026-06-22T13:17:28Z

I am not convinced this should merge before we know which implementation contract it is serving.

A core lineage query/service model without a working implementation risks freezing API concepts before we have validated the actual persistence/query semantics. In particular, the local-store discussion is still open around reduced graph semantics, “latest” edge replacement, auth-filtered reads, and backend capability boundaries.

If this PR is intended only as neutral scaffolding, I think it should be very explicit about what is not being committed here, and avoid introducing model/API shapes that later persistence PRs depend on as settled. Otherwise we should probably review this together with the first implementation that proves the model is sufficient.

Hi @snazy , thanks for clarifying. Same as #4827 I’ll continue following the dev@ discussion and update the PR based on the resulting consensus.

flyingImer · 2026-06-24T22:10:06Z

+package org.apache.polaris.lineage;
+
+/** Service boundary for lineage operations used by transport-layer adapters. */
+public interface LineageService {


confused: how is this different from PR4826?

PR #4826 adds the lineage persistence SPI and related persistence/ingest models, including PolarisLineageHandler, LineageStoreManager,LineageDataset, LineageColumnEdge, LineageEdge,LineageFieldReference, and LineageIngestRequest.

dimas-b · 2026-06-30T02:22:19Z

-  suites {
-    val intTest by registering(JvmTestSuite::class)
+/** Normalized response model for lineage queries. */
+public record LineageGraph(


This class and some others overlap with #4826 ... I'm a bit confused about the intended merge order 😅 What is the PR to be merge first?

If there are dependencies, it would be nice to keep dependent PRs in the "draft" state for the sake of clarity.

Thanks for your notification! This is the first PR and #4826 is the second. And I will also add the order at the begin of PR description and turn the others into draft.

dimas-b · 2026-06-30T14:26:07Z

+          "Lineage is disabled: set polaris.lineage.enabled=true to enable it.");
+    }
+
+    if (!callContext.getRealmConfig().getConfig(FeatureConfiguration.ENABLE_LINEAGE)) {


Why not inject just the RealmConfig (reduced dependency surface)?

dimas-b · 2026-06-30T14:27:21Z

@@ -26,6 +26,7 @@ plugins {

 dependencies {
  implementation(project(":polaris-core"))
+  implementation(project(":polaris-extensions-lineage"))


Cross-post from #4826 (comment)

I believe runtime/server should have this as a runtimeOnly dep.

The "service" module should not have hard dependencies (tests are ok) on additional API/feature modules. This is to avoid forcing new feature into all downstream projects (which have to depend on runtime/service).

iting0321 added 5 commits May 18, 2026 13:18

add Core SPI and domain model

b9265f0

update the node type

d630a09

add lineage data field

c073aae

fix the DefaultLineageServiceTest.java for feature is dark coverage

b85e227

refactor LineagePersistence

c8bd140

Copilot AI review requested due to automatic review settings June 11, 2026 13:17

github-project-automation Bot added this to Basic Kanban Board Jun 11, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jun 11, 2026

Copilot AI reviewed Jun 11, 2026

View reviewed changes

iting0321 added 2 commits June 11, 2026 21:34

remove jakarta.annotation.Nullable

a793842

add lineage feature configurations to documentation

00c432a

remove persistence-related model

bf4c5d9

dimas-b reviewed Jun 19, 2026

View reviewed changes

Move lineage types to separate module

1f17814

flyingImer reviewed Jun 24, 2026

View reviewed changes

iting0321 added 3 commits June 26, 2026 21:30

move lineage SPI models to extensions module

db746bb

move lineage handler SPI into extension module

b79394c

remove config-doc

1e3400d

dimas-b reviewed Jun 30, 2026

View reviewed changes

This was referenced Jun 30, 2026

OpenLineage: Add Persistence schema and JDBC models #4827

Draft

OpenLineage: add Persistence Layer SPI #4826

Draft

dimas-b reviewed Jun 30, 2026

View reviewed changes

Uh oh!

Conversation

iting0321 commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Out of Scope

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adnanhemani commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iting0321 commented Jun 12, 2026

Uh oh!

dimas-b Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

iting0321 Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snazy commented Jun 22, 2026

Uh oh!

iting0321 commented Jun 22, 2026

Uh oh!

flyingImer Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

iting0321 Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

iting0321 Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

iting0321 commented Jun 11, 2026 •

edited

Loading

adnanhemani commented Jun 12, 2026 •

edited

Loading

iting0321 Jun 22, 2026 •

edited

Loading